Tracking player role using non-rigid formation priors

ABSTRACT

Approaches are described for assigning roles to agents in a group of agents engaging in an activity. An assignment analysis system receives a first set of detections, where each detection in the first set of detections comprises a physical location. The assignment analysis system defines an exemplar formation comprising an arrangement of each role in a set of roles. The assignment analysis system calculates a first cost function between at least one detection in the first set of detections and at least one role in the set of roles. The assignment analysis system generates a first set of permutations based on the first cost function. The assignment analysis system assigns a first role in the set of roles to a first detection in the first set of detections based on the first set of permutations.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to artificial intelligence systems used toanalyze participants engaged in an activity and, in particular, totracking player role using non-rigid formation priors.

2. Description of the Related Art

Vision-based systems have been deployed to detect and track playersengaged in adversarial team sports. Such vision-based systems may recordthe position of each player multiple times per second over the course ofplay. For example, a vision-based system may generate location data foreach player at a rate of thirty times per second. These systems maysupport applications configured to perform various analysis functions,including, characterizing offensive patterns, recognizing individualplays, and predicting the evolution of play in a game or match.Typically, such systems characterize team behavior using a macroscopicrepresentation of the team, such as the position of the centroid of theteam players or the distribution pattern of the team players on theplaying field, or analyzing a single player over time. Suchrepresentations fail to characterize how individual players or teamsperform over time, particularly if the role of one or more playerschanges as play progresses.

In addition, tracking data for players or teams may exhibit highdimensionality, where the quantity of samples collected over longperiods of play may be too numerous for applications to efficientlyanalyze and produce reasonable results. For example, a given set oftracking data may include 200,000 frames of location data from eightdifferent camera angles. Certain temporal analyses may becomputationally prohibitive where the tracking data exhibits highdimensionality. Finally, vision systems often do not provide perfecttracking, resulting in false detections or missed detections. Analysisapplications relying on tracking data that includes such falsedetections or missed detections may produce erroneous results.Alternatively, the tracking data may be manually edited to remove anyanomalies associated with false or missed detections. However, manuallyediting tracking data is tedious and time-consuming.

SUMMARY OF THE INVENTION

One embodiment of the present invention sets forth a method forassigning roles to agents in a first group of agents engaging in anactivity. The method includes receiving a first set of detections,wherein each detection in the first set of detections comprises aphysical location. The method further includes defining an exemplarformation comprising an arrangement of each role in a set of roles. Themethod further includes calculating a first cost function between atleast one detection in the first set of detections and at least one rolein the set of roles. The method further includes generating a first setof permutations based on the first cost function. The method furtherincludes assigning a first role in the set of roles to a first detectionin the first set of detections based on the first set of permutations.

Other embodiments include, without limitation, a computer-readablemedium that includes instructions that enable a processing unit toimplement one or more aspects of the disclosed methods. Otherembodiments include, without limitation, a subsystem that includes aprocessing unit configured to implement one or more aspects of thedisclosed methods as well as a computing system configured to implementone or more aspects of the disclosed methods.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the inventioncan be understood in detail, a more particular description of theinvention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 depicts one architecture of a system within which embodiments ofthe present invention may be implemented;

FIG. 2 illustrates a playing field where the players on a team are in aformation, according to one embodiment of the present invention;

FIG. 3 illustrates a set of identity and role assignments for a subsetof team players, according to one embodiment of the present invention;

FIG. 4 illustrates a formation, a set of player detections, and aformation-detection mapping, according to one embodiment of the presentinvention;

FIG. 5 illustrates a set of formations with varying levels of areaoverlap, according to one embodiment of the present invention;

FIG. 6 illustrates a trellis graph that represents role optimization formultiple detection frames, according to one embodiment of the presentinvention;

FIG. 7 sets forth a flow diagram of method steps for assigning playersto detection information, according to one embodiment of the presentinvention;

FIGS. 8A-8B set forth a flow diagram of method steps for assigningplayers to detection information, according to another embodiment of thepresent invention.

DETAILED DESCRIPTION

In the following, reference is made to embodiments of the invention.However, it should be understood that the invention is not limited tospecific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theinvention. Furthermore, although embodiments of the invention mayachieve advantages over other possible solutions and/or over the priorart, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the invention. Thus, the followingaspects, features, embodiments and advantages are merely illustrativeand are not considered elements or limitations of the appended claimsexcept where explicitly recited in a claim(s). Likewise, reference to“the invention” shall not be construed as a generalization of anyinventive subject matter disclosed herein and shall not be considered tobe an element or limitation of the appended claims except whereexplicitly recited in a claim(s).

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. Such computer, other programmable apparatus orother device may include, without limitation, a personal computer, videogame console, personal digital assistant, rendering engine, mobiledevice, or dedicated hardware platform, such as a very large scaleintegrated (VLSI) circuit, a field-programmable gate array (FPGA), or anapplication specific integrated circuit (ASIC).

Hardware Overview

FIG. 1 depicts one architecture of a system 100 within which embodimentsof the present invention may be implemented. This figure in no waylimits or is intended to limit the scope of the present invention.

System 100 may be a personal computer, video game console, personaldigital assistant, rendering engine, or any other device suitable forpracticing one or more embodiments of the present invention.

As shown, system 100 includes a central processing unit (CPU) 102 and asystem memory 104 communicating via a bus path that may include a memorybridge 105. CPU 102 includes one or more processing cores, and, inoperation, CPU 102 is the master processor of system 100, controllingand coordinating operations of other system components. System memory104 stores software applications and data for use by CPU 102. CPU 102runs software applications and optionally an operating system. Memorybridge 105, which may be, e.g., a Northbridge chip, is connected via abus or other communication path (e.g., a HyperTransport link) to an I/O(input/output) bridge 107. I/O bridge 107, which may be, e.g., aSouthbridge chip, receives user input from one or more user inputdevices 108 (e.g., keyboard, mouse, joystick, digitizer tablets, touchpads, touch screens, still or video cameras, motion sensors, and/ormicrophones) and forwards the input to CPU 102 via memory bridge 105. Inone embodiment, the computer system 100 is configured to implement anassignment analysis system that may receive player detection locationdata and assign player identities and roles to the player detectionlocation data, as further described herein.

A display processor 112 is coupled to memory bridge 105 via a bus orother communication path (e.g., a PCI Express, Accelerated GraphicsPort, or HyperTransport link); in one embodiment display processor 112is a graphics subsystem that includes at least one graphics processingunit (GPU) and graphics memory. Graphics memory includes a displaymemory (e.g., a frame buffer) used for storing pixel data for each pixelof an output image. Graphics memory can be integrated in the same deviceas the GPU, connected as a separate device with the GPU, and/orimplemented within system memory 104.

Display processor 112 periodically delivers pixels to a display device110 (e.g., a screen or conventional CRT, plasma, OLED, SED or LCD basedmonitor or television). Additionally, display processor 112 may outputpixels to film recorders adapted to reproduce computer generated imageson photographic film. Display processor 112 can provide display device110 with an analog or digital signal.

A system disk 114 is also connected to I/O bridge 107 and may beconfigured to store content and applications and data for use by CPU 102and display processor 112. System disk 114 provides non-volatile storagefor applications and data and may include fixed or removable hard diskdrives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, orother magnetic, optical, or solid state storage devices.

A switch 116 provides connections between I/O bridge 107 and othercomponents such as a network adapter 118 and various add-in cards 120and 121. Network adapter 118 allows system 100 to communicate with othersystems via an electronic communications network, and may include wiredor wireless communication over local area networks and wide areanetworks such as the Internet.

Other components (not shown), including USB or other port connections,film recording devices, and the like, may also be connected to I/Obridge 107. For example, an audio processor may be used to generateanalog or digital audio output from instructions and/or data provided byCPU 102, system memory 104, or system disk 114. Communication pathsinterconnecting the various components in FIG. 1 may be implementedusing any suitable protocols, such as PCI (Peripheral ComponentInterconnect), PCI Express (PCI-E), AGP (Accelerated Graphics Port),HyperTransport, or any other bus or point-to-point communicationprotocol(s), and connections between different devices may use differentprotocols, as is known in the art.

In one embodiment, display processor 112 incorporates circuitryoptimized for graphics and video processing, including, for example,video output circuitry, and constitutes a graphics processing unit(GPU). In another embodiment, display processor 112 incorporatescircuitry optimized for general purpose processing. In yet anotherembodiment, display processor 112 may be integrated with one or moreother system elements, such as the memory bridge 105, CPU 102, and I/Obridge 107 to form a system on chip (SoC). In still further embodiments,display processor 112 is omitted and software executed by CPU 102performs the functions of display processor 112.

Pixel data can be provided to display processor 112 directly from CPU102. In some embodiments of the present invention, instructions and/ordata representing a scene are provided to a render farm or a set ofserver computers, each similar to system 100, via network adapter 118 orsystem disk 114. The render farm generates one or more rendered imagesof the scene using the provided instructions and/or data. These renderedimages may be stored on computer-readable media in a digital format andoptionally returned to system 100 for display. Similarly, stereo imagepairs or multiview autostereoscopic images processed by displayprocessor 112 may be output to other systems for display, stored insystem disk 114, or stored on computer-readable media in a digitalformat.

Alternatively, CPU 102 provides display processor 112 with data and/orinstructions defining the desired output images, from which displayprocessor 112 generates the pixel data of one or more output images,including characterizing and/or adjusting the offset between stereoimage pairs, in the case of stereoscopic images, or generating andinterleaving multiple views, in the case of multiview autostereoscopicimages. The data and/or instructions defining the desired output imagescan be stored in system memory 104 or graphics memory within displayprocessor 112. For example, CPU 102 could execute a client media playerapplication (not shown) that receives a media stream from a contentprovider, and transmits the media stream to the display processor 112for viewing on the display device 110. In an embodiment, displayprocessor 112 includes 3D rendering capabilities for generating pixeldata for output images from instructions and data defining the geometry,lighting shading, texturing, motion, and/or camera parameters for ascene. Display processor 112 can further include one or moreprogrammable execution units capable of executing shader programs, tonemapping programs, and the like.

CPU 102, render farm, and/or display processor 112 can employ anysurface or volume rendering technique known in the art to create one ormore rendered images from the provided data and instructions, includingrasterization, scanline rendering REYES or micropolygon rendering, raycasting, ray tracing, image-based rendering techniques, and/orcombinations of these and any other rendering or image processingtechniques known in the art.

It will be appreciated that the system shown herein is illustrative andthat variations and modifications are possible. The connection topology,including the number and arrangement of bridges, may be modified asdesired. For instance, in some embodiments, system memory 104 isconnected to CPU 102 directly rather than through a bridge, and otherdevices communicate with system memory 104 via memory bridge 105 and CPU102. In other alternative topologies display processor 112 is connectedto I/O bridge 107 or directly to CPU 102, rather than to memory bridge105. In still other embodiments, I/O bridge 107 and memory bridge 105might be integrated into a single chip. The particular components shownherein are optional; for instance, any number of add-in cards orperipheral devices might be supported. In some embodiments, switch 116is eliminated, and network adapter 118 and add-in cards 120, 121 connectdirectly to I/O bridge 107.

Representing and Discovering Adversarial Team Behavior Using PlayerRoles

Techniques are presented for representing and discovering adversarialgroup behavior by assigning roles to group members. Although presentedin the context of team sports that take place on a playing field, thedescribed techniques as applied to any technically feasible environmentwhere similar behavior is present are within the scope of thisdisclosure.

In comparison to other types of behavior, adversarial behavior isheavily structured in that the location of a player or, more generally,an agent, is dependent both on the player's teammates and adversaries,in addition to the tactics or strategies of the team. The describedtechniques may take advantage of this behavioral structure through theuse of a spatiotemporal basis model. Players may change roles multipletimes during a game or match. For example, a first player with the roleof center forward in field hockey could swap roles with a second playerwith the role of right wing. After swapping roles, the first playerwould have the role of right wing, while the second player would havethe role of center forward. Accordingly, employing a “role-based”representation, rather than a representation based on player “identity,”may better represent the playing structure of the team. In addition,vision-based systems generally do not provide perfectdetection/tracking, resulting in missed or false detections. Thedescribed techniques may “clean” the tracking data to compensate forsuch missed or false detections prior to assigning player roles, whichin turn enables analysis of team formation and plays as may occur duringcontinuous-play sports. The disclosed techniques also describe anapproach to reduce the memory consumed by tracking data by convertingthe raw tracking data into a representation based on shape basis andtrajectory basis. Such an approach takes advantage of the temporalsmoothness of human motion to represent tracking data to reduce theamount of data to accurately reflect team player motion. As a result,the converted tracking data is stored and processed more efficiently, ascompared with raw tracking data, enabling more significant temporalanalysis by applications.

A group of individuals occupying a space, such as a crowd in a foyer ora gathering at a public square, may opportunistically exhibitrecognizable patterns of interaction. For example, individuals may moveso as to avoid collisions with each other or with structuralconstraints, such as lamp-posts. By contrast, for individuals in a teamcompetitive environment, such as during games on a sports field,distinct and deliberate patterns of activity emerge in the form ofplays, tactics, and strategies. In the former case, each individualpursues an individual goal based on an individual schedule. In thelatter case, teams engage in adversarial goal-seeking, typically(although not necessarily) under the synchronized direction of a captainor a coach. Identifying such emergent patterns of play may aid fans,players, coaches, and broadcasters (including commentators, cameraoperators, producers, and game statisticians) in understanding as thegame evolves and progresses.

The behavior of a team may be described by how team members cooperateand contribute in various situations. In team sports, the overall styleof a team may be characterized by a formation, where a formation is acoarse spatial structure that the players maintain over the course ofthe match. Additionally, player movements are governed by physicallimits, such as acceleration, resulting in smooth player trajectoriesover time. These two observations suggest significant correlation, andtherefore redundancy, in the spatiotemporal signal of player movementdata. This correlation may be exploited to generate an approximation ofthe spatiotemporal behavior of the players, while retaining sufficientaccuracy to facilitate analysis. This approximation may aid applicationsin analysis that may improve understanding of team behavior. First,recovering an approximation of player behavior may enable the recoveryof a true underlying signal from a set of player tracking data that mayinclude false or missing detections. Second, these approximations mayincrease the ability of applications to recognize previously observedgame situations in new data.

Even perfect tracking data may not be sufficient for understanding teambehavior. A formation implicitly defines a set of roles or individualresponsibilities which are distributed among the players by the captainor coach. In dynamic games like soccer or field hockey, players mayopportunistically swap roles, either temporarily or permanently. As aresult, when analyzing the strategy of a particular game situation,players are typically identified by the role the players are currentlyplaying and not necessarily by an individualistic attribute like thename of the player. The disclosed techniques may be used to analyzedetection and tracking data based on the role of each player at anygiven time, rather than by the identity of the player. Associating rolesto player locations may provide a more compact low-dimensionalrepresentation, allowing the use of a bilinear spatiotemporal model thatenables removal of tracking data “noise,” such as errors caused by falseor missing detections common in many vision systems. The compactlow-dimensional representation may aid applications in identifyingformations and plays quickly from a large repository, thus enhancingsports commentary by highlighting recurrent team strategies and longterm trends in a sport. In addition, the process of post-gameannotation, typically performed manually by coaches and technical staffover a period of many hours, may be automated. Such automaticallyannotated tracking data may enable applications to efficiently performmore detailed data mining. Such analysis may be employed by anautomation system to predict motion on the field, where the predictedmotion may direct robotic cameras to follow the action on the field.

FIG. 2 illustrates a playing field 200 where the players 210 on a teamare in a formation, according to one embodiment of the invention. Asshown, the players are in a 5-3-2 formation where five players are infield quadrant 240, three players are in field quadrant 230, and twoplayers are in field quadrant 220.

In many team sports, a coach or captain designates an overall structureor system of play for a team. For example, in field hockey, thestructure could be described as a formation involving roles orindividual responsibilities for each player on the team. In the 5:3:2formation illustrated in FIG. 2, the defined set of roles includes leftback (LB), right back (RB), left halfback (LH), center halfback (CH),right halfback (RH), left wing (LW), inside left (IL), center forward(CF), inside right (IR), and right wing (RW), as assigned to players210(0)-210(9) respectively. Each player is assigned exactly one role,and every role is assigned to only one player. Generally, roles are notfixed. During a match, players may swap roles and temporarily adopt theresponsibilities of another player, typically to exploit a tacticalopportunity or to respond to behavior of the adverse team. As shown inFIG. 2, player 210(2) may move to a position the outside of player210(5), taking the role of left wing. In response, player 210(6) maymove toward the prior position of player 210(2), taking the role of lefthalfback. Player 210(5) then assumes the role of inside left.

A player tracking system may generate a series

of observations, where each observation includes the (x, y) position ofeach player on the field, a team estimate

ε{α,β} for each player, and a time stamp t. At any given time instant t,the set of detected player locations

_(t)={x_(A), y_(A), x_(B), y_(B), . . . } is of arbitrary length. Thatis, the number of detections N_(t) at time t may not necessarily beequal to the number of players P on a team because some players may nothave been detected, background noise may have been incorrectlyclassified as a player, or a player may be off-field due to a penalty.

Typically, the goal is to track all 2P players over the duration of thematch. In field hockey, that corresponds to tracking 20 players, whereP=10 players per team excluding the goalkeepers, over two halves of 35minutes each. The tracking all players across time may be expressed as avector of ordered player locations

=[x₁, y₁, x₂, y₂, . . . , x_(P), y_(P)]^(T) for each team

from the potentially noisy detections

_(t) at each time instant. Although the particular ordering of playersmay be arbitrary, the ordering is consistent across time. As such,

may be considered to be a static labeling of player locations. Note that

is not generally a subset of

_(t). For example, if a player is not detected at a given observationtime, the (x, y) position of the undetected player may be inferred basedon spatiotemporal correlations. Any observed arrangement of players fromthe first team α may also be observed for players of the second team β,but in the opposing direction. As such, team β observation data mayexhibit a 180° symmetry with respect to team α observation data. Thatis, for any given vector of player locations

, there is an equivalent complementary vector

that may be derived by rotating all (x, y) locations about the center ofthe field and swapping the associated team affiliations.

As described herein, a coach or captain may designate an overallstructure, or formation, that establishes roles or individualresponsibilities for each team player. Mathematically, assigning rolesto players is equivalent to permuting the player ordering

. A permutation matrix

, of size P×P, may be defined at time t, where the permutation matrixdescribes the players in terms of roles

, according to Equation 1 below:

=

  Equation 1where each element

(i, j) of the permutation matrix

is a binary variable with value 0 or 1, and each column and row of thepermutation matrix

sums to 1. Accordingly, if

(i, j)=1, then player i is assigned to role j. As such, whereas theordered player locations

is considered to be a static labeling of player locations, the set ofroles

, is a dynamic labeling of player locations.

Because the spatial relationships of a formation are defined in terms ofroles, and not in terms of individualistic attributes like player nameor jersey number, and because players swap roles during a game, thespatiotemporal patterns based on the set of roles {

,

, . . . . ,

} may be more compact as compared with the spatiotemporal patterns basedon individual players {

,

, . . . . ,

}. In addition, because a team may be expected to maintain the sameformation (set of roles) as the team moves up the field and down thefield, the player position data

relative to the mean (x, y) location of the team may be morecompressible as compared with the absolute player position data

.

A player's movements are correlated not only to the positions and rolesof teammates but to the positions and roles of opposition players aswell. As such, player location data may be further compressed if thelocations of players on teams A and B are concatenated into a singlevector p_(t) ^(AB)=[p_(t) ^(A), p_(t) ^(B)]^(T), referred to asadversarial representation. Employing an adversarial representation ofplayer locations may achieve better compressibility than using onlyordered player,

, or role,

, position data of one team. Using both a role and adversarialrepresentation together, r_(t) ^(AB)=[r_(t) ^(A), r_(t) ^(B)]^(T), mayyield more compressibility than using either role representation orordered player based adversarial representation alone.

A bilinear spatiotemporal basis model that captures and exploits thedependencies across both the spatial and temporal dimensions in anefficient manner may be applied to the location data described herein.Given P players per team, or 2P total players, sampled at F timeinstances, the role-based adversarial representation x may be formed asa spatiotemporal structure S, according to equation 2 below:

$\begin{matrix}{S_{F \times 2P} = \begin{bmatrix}x_{1}^{1} & \ldots & x_{2P}^{1} \\\vdots & \; & \vdots \\x_{1}^{F} & \ldots & x_{2P}^{F}\end{bmatrix}} & {{Equation}\mspace{14mu} 2}\end{matrix}$where x_(j) ^(i) denotes the jth index within the role representation atthe ith time instant. Accordingly, the time-varying structure matrix Sincludes 2FP parameters. This representation of the structure S is anover-parameterization in that the representation does not take intoaccount the high degree of regularity generally exhibited by motiondata. In one embodiment, this regularity in spatiotemporal data may beexploited by representing the 2D formation or shape exhibited by theplayers on a team at each time instance as a linear combination of asmall number of shape basis vectors b_(j) weighted by coefficients ω_(j)^(i) as s^(i)=Σ_(j)ω_(j) ^(i)b_(j) ^(T). Alternatively, the time-varyingstructure may be represented by modeling the representation in thetrajectory subspace, as a linear combination of trajectory basis vectorsθ_(i) as s_(j)=Σ_(i)α_(i) ^(j)θ_(i), where α_(i) ^(j) is the coefficientweighting each trajectory basis vector. As a result, the completestructure matrix may be represented by Equation 3 below:S=ΩB ^(T)  Equation 3where Ω is a F×K_(s) matrix that includes the corresponding shapecoefficients ω_(j) ^(i), and B is a P×K_(s) matrix that includes K_(s)shape basis vectors, each of which represents a 2D structure of length2P. Alternatively, the complete structure matrix may be represented byEquation 4 below:S=θA ^(T)  Equation 4where Θ is a F×K_(t) matrix that includes K_(t) shape basis vectorstrajectory bases as the columns, and A is a 2P×K_(t) matrix oftrajectory coefficients. The quantity of shape basis vectors used torepresent a particular instance of motion data is represented byK_(s)≦min{F, 2P}, and K_(t)≦{F, 2P} is the quantity of trajectory basisvectors spanning the trajectory subspace.

Both representations of S are over-parameterizations in that therepresentations do not capitalize on either the spatial or temporalregularity exhibited in the data. Because S may be expressed exactlyeither by Equation 3 or by Equation 4, then there exists a factorizationas shown in Equation 5 below:S=ΘCB ^(T)  Equation 5where C=Θ^(T)Ω=A^(T)B is a K_(t)×K_(s) matrix of spatiotemporalcoefficients. Equation 5 describes the bilinear spatiotemporal basis,which includes both shape and trajectory bases linked together by acommon set of coefficients.

Due to the high degree of temporal smoothness in the motion of humans, apredefined analytical trajectory basis may be used without significantloss in representation accuracy. In one embodiment, the conditioningtrajectory basis may be a Discrete Cosine Transform (DCT) basis. The DCTbasis may be close to the optimal Principal Component Analysis (PCA)basis if the location data is generated from a stationary first-orderMarkov process. Given the high temporal regularity typically present inhuman motion, the DCT basis may be particularly appropriate fortrajectories associated with human faces and bodies. Due to the highlystructured nature of typical adversarial team sports, and the fact thathuman motion is relatively simple when measured over short periods oftime, significant dimensionality reduction may be achieved, particularlyin the temporal domain. For example, five-second plays could beeffectively represented with no more than K_(t)=3 and K_(s)=33 with amaximum error of less than two meters. In terms of dimensionalityreduction, temporal signals could be represented using 3×33=99coefficients, resulting in a 60:1 reduction in dimensionality. Greatercompressibility could be achieved where plays are longer than fiveseconds.

Roles may be assigned automatically to the arbitrary order of playerlocations

.

Assuming that a prototype formation with role ordering exists, which isdenoted as r_(proto) ^(T), the optimal assignment of roles may bedefined as the permutation matrix x_(t) ^(T)* that minimizes the squareL₂ reconstruction error as expressed in Equation 6 below:

$\begin{matrix}{x_{t}^{T\;\bigstar} = {\arg\;{\min\limits_{x_{t}^{T}}{{r_{proto}^{T} -}}_{2}^{2}}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

Equation 6 represents a linear assignment problem, where an entry C(i,j) in the cost matrix represents the Euclidean distance between rolelocations as shown in equation 7 below:C(i,j)=∥r _(proto) ^(T)(i)−

(j)∥₂  Equation 7

In one embodiment, an optimal permutation matrix may be found inpolynomial time using the Hungarian, or Kuhn-Munkres algorithm, asfurther described herein.

As a starting point, a reference formation is selected that representsthe mean starting formation of the team, such as a formation that iscommonly used by the team at the beginning of a segment of play. Such areference formation may be predetermined or may be selected via anytechnically feasible approach, including, without limitation, selectingthe reference formation from a codebook, selecting the referenceformation from a set of formations learned from prior tracking data, orreceiving the reference formation as an input from a user. As playprogresses, subtle changes in the formation may occur for variousreasons, including, without limitation, the behavior of the opposingteam and the current state of the game.

In some embodiments, these formation changes are incorporated byselecting new formations from a codebook of possible formations that theteam may employ. The formations from the codebook may be mapped to a setof “training data” where the training data includes sample location datafrom play segments examples that have been identified with assignmentlabels corresponding to the named formations from the codebook.

As such, a mapping matrix W may be learned by an analysis application bycomparing the mean and covariances of the training data with theassignment labels corresponding to the training data. Given N trainingsegment examples, the mapping matrix W may be learned by concatenatingthe mean and covariance into an input vector z_(n) corresponding to thelabeled formation x_(n) from the codebook. The resulting vectors may becompiled into grand profile matrices X and Z. Given these grand profilematrices, linear regression may be employed to learn the mapping matrixW by solving Equation 8 below:W=XZ ^(T)(ZZ ^(T) +λI)⁻¹  Equation 8where λ is a regularization term. Using this approach, a labeledformation may be estimated from the training data set that bestdescribes the current unlabeled data set. This approach may produce moreaccurate assignment performance as compared with using the meanformation for both player identity and role labels.

In practice, vision-based player detection systems do not generateperfect player position data. As such, the techniques described hereininterpret “noisy” player location data from such systems, where noisydata may include missed and false detections.

In one embodiment, a vision based player detection system may generateframes of player location data at a fairly high rate, such as 30 framesper second, and may simultaneously analyze video data from multiplecamera sources. The vision based system may determine player positionsby subtracting background information representing the field (i.e., thestatic part of the captured scene) from the received video data andemploying a coarse 3D geometric model of a person to identify theposition of each player on the field. Once the locations of all playersare determined, players may be classified as members of the respectiveteams by using a color model for each team that, for instance,represents the colors of the uniforms for each team. Each player imagemay be represented as a histogram in an appropriate color space,including, without limitation, LAB color space, CIE color space, or RGBcolor space. A generalized model for each team may be learned by usingan approach involving K-means clustering using the Bhattacharyyadistance. A player may be considered as detected if the player is withintwo meters of a ground-truth label.

As a starting point, the majority of player detections and teamaffiliations as determined by the player detection system may be assumedto be correct. Each player detection is assigned a label if thedetection is deemed to be accurate, or discarded if the detection isdeemed to be too noisy. To determine whether or not a detection shouldbe assigned a label or discarded, some feature of the game context isdetermined, such as the section of the field where most of the teamplayers are positioned. An approach, such as the approach describedabove to assign roles to players, is employed to determine this gamecontext. However, rather than learning the mapping from the cleanfeatures Z of the team formation, as in the case of role assignment, themapping is learned from the noisy features Z_(noisy). Such noisyfeatures may result from “black-spots” on the field where a portion ofthe field is not visible to any of the cameras, either based on thecurrent camera positions or due to a concentration of players in aportion of the field. Such black spots may result in a missed detection.Accordingly, the noisy feature context Z_(noisy) includes the quantityof players detected from the player tracking system as well as mean andcovariance information. The assignment analysis system then learns thenoisy mapping matrix W_(noisy) from the noisy feature context Z_(noisy).In doing so, the assignment analysis system may assume that the cleancentroid team position is a good approximation of the noisy centroidteam position. With this assumption, the assignment analysis system mayselect a reasonable prototypical formation to make player assignments.

In one embodiment, the assignment analysis system may use the estimatedprototype formation to assign players to roles by using the so-called“Hungarian algorithm,” where the Hungarian method is an optimizationalgorithm that resolves assignment problems by determining a lowest costpath. Missed and player false detections may alter the one-to-onemapping between the prototype formation and the input detections, whichmay lead to erroneous assignments. To resolve missed and falsedetections, an exhaustive approach may be used, such that if fewerplayer detections are received than the quantity of players in theprototype formation, then the assignment analysis system may determineall possible role assignments for the detected players, and then selectsthe combination that yields the lowest cost. If more player detectionsare received than the quantity of players in the prototype formation,then the assignment analysis system may determine all possiblecombinations that the detections could be and then may select thecombination of detections with the lowest cost.

For example, if nine players are detected for a prototype formation thatincludes ten players, then the assignment analysis system could computethe ten possible combinations that the detected players could be mappedto the prototype formation, namely: [2, 3, 4, . . . , 10], [1, 3, 4, . .. , 10], [1, 2, 4, . . . , 10], [1, 2, 3, 5, . . . , 10], . . . , [1, 2,3, . . . , 9]. The assignment analysis system could perform theHungarian algorithm for each of these combinations and calculates thecost of the potential assignments. After the cost of each combination iscalculated, the assignment analysis system would select the potentialassignment that yields the lowest cost. If eleven players are detectedfor a prototype formation that includes ten players, then the assignmentanalysis system could compute each of the eleven possible combinations,where each combination includes ten of the eleven detections. Theassignment analysis system could then calculate the cost of each of thepossible combinations and would select the combination with the lowestcost. Even if ten detections are received for a ten-player prototypeformation, some detections could be false positives, where only seven oreight of the ten detections represent valid candidate player positions.In such cases, the assignment analysis system could filter suchpotential false detections. The assignment analysis system could removeplayer detections that are more than a threshold distance, such astwenty meters, from the nearest other player.

Employing this approach may improve the assignment precision rate, whilethe recall rate may decrease, where the precision rate is the fractionof retrieved instances that are relevant and the recall rate is thefraction of relevant instances that are retrieved. In one example, a setof detections for ten players could include a total of seven actualdetections where one of the seven detections is a false positive.Because six of the ten players are accurately detected, the recall ratewould be 6/10 or 60%. Because six of the seven detections are truepositives, the precision is 6/7 or approximately 86%. If, after afiltering operation, the false positive and one of the true positives isremoved from the set of detections, then the set of detections wouldinclude five a total of five actual detections with no false positives.Accordingly, the recall rate would decrease to 5/10 or 50% while theprecision rate would increase to 5/5 or 100%. However, even with areduced recall rate, role assignments using these techniques may beimproved over using raw player position data.

The assignment analysis system computes a continuous estimate of theplayer label at each time step over a period of time by temporallysmoothing the data computed at each time step. This continuous estimateis then used for further formation and play analysis.

To do so, the assignment analysis system may perform an expectationmaximum (EM) process, using the spatial bases, the bilinear coefficientsand an initial estimate of the player labels as inputs to the process.In one embodiment, the expectation phase of the EM process may besimplified to making an initial assignment of the player labels, whichmay be determined from the initial assignments calculated using theapproach described above. From this initial assignment, an initial valuefor S_(init) may be determined. During the maximization phase of the EMprocess, the value C may be calculated as C=θ^(T)S_(init)B. The finalvalue S_(final) may then be estimated from the calculated value of C,the spatial basis B and the temporal basis θ.

Tracking Players Using Non-Rigid Formation Priors

As described above, players may be assigned to roles based on playerdetection data and a prototype formation, such as a formation selectedfrom a code book. As play progresses, players may instantaneously swaproles. Such players may not necessarily be next to each other when theplayers swap roles. As such, while the players themselves may move alimited distance between position detection frames, the set of roles donot generally have such spatiotemporal constraints. Player identitiesand roles may be assigned to unlabeled (x, y) position data, whereplayer identity may be assigned based on spatiotemporal constraints,while player roles may be assigned based on only spatial properties,using absolute position and position relative to other players. Roleschange may be assumed to be infrequent, such that the spatial prior of aformation may be combined with the spatiotemporal prior of playerinertia to track player identities and roles over time. Such identityand role assignments may be applied throughout a game or match andcompared with similar data from other matches from a current or priorseason. Such an approach may enhance game analysis both during a game ormatch and after completion of the game or match.

Once roles are assigned to player detections at a particular timeinstance, as described above, identities and roles may be assigned to asequence of detections over time, where the assignment analysis systemconsiders not only the likelihood of per-frame identity and roleassignments, but also the temporal consistency of these assignmentsacross sequential detection frames.

FIG. 3 illustrates a set of identity and role assignments 300 for asubset of team players, according to one embodiment of the presentinvention. As shown, the set of identity and role assignments 300includes three players 310, 320, 330 in an initial position 302, apotential second position 304, and an alternate potential secondposition 306.

The initial position 302 includes three players 310(0), 320(0), 330(0),where each player is associated with a triplet I:L:R, where ‘I’ is anarbitrary index from the player detection system, ‘L’ is a labelassigned to identify the player, and ‘R’ is a role assigned to theplayer.

As shown, the index I is a numerical value that the player detectionsystem arbitrarily assigns to each detection in a given detection frame.As such, the index I for a given player may vary arbitrarily from onedetection frame to the next. The array of indices for the three playersin the initial position 302 at a given time t shown may be expressed byd_(t)=[1, 2, 3]^(T).

The label L is a label assigned to each player to identify a particularplayer. The label may be any technically feasible characteristic that isattached to a particular player, including, without limitation, a playername, a jersey number, or other identifying characteristic. As shown,the label L is a single letter assigned to each individual player. Thearray of labels for the three players in the initial position 302 at agiven time t shown may be expressed by

_(t)=[A, B, C]^(T). The label of a player may also be referred to as theidentity of the player.

The role R is the player position or role assigned to each player. Forexample, the role R could be an abbreviation for the position of eachplayer. The array of labels for the three players in the initialposition 302 at a given time t shown may be expressed by

=[LF, CB, RF]^(T), corresponding to the roles of left forward, centerback, and right forward, respectively.

Player 310(0) is associated with the triplet 2:C:LF, indicating that theplayer 310(0) is identified as player C having the role of left forward,and being the second player detection at the initial position 302.Player 320(0) is associated with the triplet 3:B:RF, indicating that theplayer 320(0) is identified as player B having the role of rightforward, and being the third player detection at the initial position302. Player 330(0) is associated with the triplet 1:A:CB, indicatingthat the player 330(0) is identified as player A having the role ofcenter back, and being the first player detection at the initialposition 302.

The potential second position 304 includes the three players 310(1),320(1), 330(1), at a time after the initial position 302. As shown, thethree players 310(1), 320(1), 330(1) have moved to new positions, whereeach player has retained the role assigned at the initial position 302.That is, player A is the center back, player B is the right forward, andplayer C is the left forward.

The alternate potential second position 306 includes the three players310(2), 320(2), 330(2), at a time after the initial position 302. Asshown, the three players 310(2), 320(2), 330(2) have moved to newpositions, and each player has moved to a new role as compared with theinitial position 302. That is, player A has advanced and moved to theright, moving from the center back role to the right forward role.Player B has moved left field, moving from the right forward role to theleft forward role. Player C has dropped back and to the right, movingfrom the left forward role to the center back role.

The team as a whole has a set of

players who are dynamically assigned responsibilities from a set of

roles, where the set of roles is generally defined by the strategyemployed by the team, and may be determined by a particular formation.Typically, each player is assigned one role, and every role is assignedto one player.

The assignment analysis system may infer the role fulfilled by eachplayer at each time instant corresponding to a detection frame. Theinput data from the player detection system is a set of detected 2Dplayer positions

_(t) at each time instant t given by the equation:

_(t)={(x, y)₁. (x, y)₂, . . . (x, y)_(N)}, where N is the number ofdetected players. Both an identity and a role as assigned to each (x, y)detection. Roles refer to players in an abstract strategic sense. Assuch, there is a one-to-one mapping between the set of identities andthe set of roles. At each time instant, each identity and each role isassigned to one detection, and each detection includes one associatedidentity and one associated role associated to it. Accordingly, twoassignment problems are simultaneously resolved for the set ofdetections, one for role assignment and another for label or identityassignment.

An assignment of roles may be represented as a permutation that shufflesthe set of player detections into role order. Likewise, an assignment ofidentities may be represented as a permutation that shuffles the set ofplayer detections into identity order.

A permutation of N elements may be represented in one of three ways: (1)as an integer sε[1, N!] indexing a particular permutation in the set

_(N) of permutations; (2) as a vector ŝ of permuted indices ranging from1 to N; or (3) as an N×N binary matrix S. As used herein, a caret accentidentifies a vector of permuted indices, as distinguished from a vectorof permutations. A set of N individual assignments encoded in apermutation may be recovered by multiplying the vector d_(t)

[1, 2, 3, . . . , N]^(T) by the corresponding permutation matrix.

Because assignments of identities and roles to the set of detections

_(t) at time t may be represented as permutations, the variables P and Rmay be defined to represent the permutations indicating the indentifyassignments and role assignments, respectively, for the time intervaltε[1, T] as shown in Equations 9 and Equation 10 below:P=[P ₁ ,P ₂ , . . . ,P _(T)]  Equation 9R=[R ₁ ,R ₂ , . . . ,R _(T)]  Equation 10

Each P_(t)ε

_(N) represents a permutation which assigns identities

at time t. Similarly, each R_(t) represents a permutation which assignsroles

at time t. A set of more probable role and identity assignments may bedetermined given a time sequence of detections: D=[

₁,

₂, . . . ,

_(T)].

The assignment of permutations (P_(t)=p_(t)) and (R_(t)=r_(t)) implies Nindividual assignments of identity and role, which may represented asvectors of permuted indices: {circumflex over (p)}_(t)=P_(t)d_(t) and{circumflex over (r)}_(t)=R_(t)d_(t), respectively.

For example, the three players of FIG. 3 may be represented by playerlabels/identities

={A, B, C}, and player roles

={LF, CB, RF}. Given three players, there exists six possiblepermutations

₃=[(1,2,3), (1,3,2), (2,1,3), (2,3,1), (3,1,2), (3,2,1)].

If the second permutation is selected for the identities assignment(P=2), then the player identity assignment may be expressed according toequation 11 below:

$\begin{matrix}{\left( {P = 2} \right) \equiv \left( {\hat{p} = \begin{bmatrix}1 \\3 \\2\end{bmatrix}} \right) \equiv \begin{matrix}\left. A\rightarrow\left( {x,y} \right)_{1} \right. \\\left. B\rightarrow\left( {x,y} \right)_{3} \right. \\\left. C\rightarrow\left( {x,y} \right)_{2} \right.\end{matrix}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

If the third permutation is selected for the roles assignment (R=3),then the player role assignment may be expressed according to equation12 below:

$\begin{matrix}{\left( {R = 3} \right) \equiv \left( {\hat{r} = \begin{bmatrix}2 \\1 \\3\end{bmatrix}} \right) \equiv \begin{matrix}\left. {LF}\rightarrow\left( {x,y} \right)_{2} \right. \\\left. {CB}\rightarrow\left( {x,y} \right)_{1} \right. \\\left. {RF}\rightarrow\left( {x,y} \right)_{3} \right.\end{matrix}} & {{Equation}\mspace{14mu} 12}\end{matrix}$

As shown in FIG. 3, player identities are given by alphabetic labels A,B, C, . . . , rather than by player name or jersey number. Accordingly,the player identities may be given by the set of alphabetic labels

={A, B, C, . . . }. Solving for player identities P may then beexpressed as solving for a vector of label variables L=[L₁, L₂, . . . ,L_(T)], where each L_(t)ε

_(N) represents a given permutation that assigns labels

to player detections. The assignment problem may then be expressed asinferring to most probable sequence of labels L=[L₁, L₂, . . . , L_(T)]and roles L=[L₁, L₂, . . . , L_(T)] to assign to a temporal sequence ofdetections

as given by equation 13 below:(l,r)*=argmaxP(L=l,R=r|D)(l,r)  Equation 13

The joint assignment ((L_(t)=l_(t))∩(R_(t)=r_(t))) of labels and rolesat time t may be represented by Y_(t)ε

_(N)×

_(N). Both labels/identities assignments and roles assignments mayexhibit a temporal Markov property and, accordingly, the problem may bemodeled as a linear-chain conditional random field. As a result, theconditional probability factors as a product of exponential functionsinvolving potential energies U_(t)(y_(t), y_(t-1),

_(t)), as given by Equation 14 below:

$\begin{matrix}{{P\left( {{L = l},{R = \left. r \middle| D \right.}} \right)} = {{P\left( {Y = \left. y \middle| D \right.} \right)} = {\frac{1}{Z(D)}{\prod\limits_{t = 1}^{T}\;{\mathbb{e}}^{- {U_{t}{({y_{t},y_{t - 1},{??}_{t}})}}}}}}} & {{Equation}\mspace{14mu} 14}\end{matrix}$

The partition function Z(D) is a normalization constant that depends onthe observed data, but does not affect the optimization process. Theassignment problem of equation 13 may then be equivalent to minimizingthe negative log likelihood as expressed by the alternative assignmentproblem of Equation 15 below:

$\begin{matrix}{y^{\bigstar} = {\underset{y}{\arg\;\min}{\sum\limits_{t = 1}^{T}{U_{t}\left( {y_{t},y_{t - 1},{??}_{t}} \right)}}}} & {{Equation}\mspace{14mu} 15}\end{matrix}$

The potential energy function at each time instant may be modeled as acollection of functions which examine the likelihood of independentassignments of labels E_(t)(l_(t),

_(t)) and roles E_(t)(r_(t),

_(t)) at each time instant, as well as independent temporal transitions,E_(t)(l_(t), l_(t-1),

_(t)) and E_(t)(r_(t), r_(t-1),

_(t)), and the joint transition E_(t)(l_(t), r_(t), l_(t-1), r_(t-1),

_(t)) as shown in Equation 16 below:U _(t)(y _(t) ,y _(t-1),

_(t))=E _(t)(l _(t),

_(t))+E _(t)(r _(t),

_(t))+E _(t)(l _(t) ,l _(t-1),

_(t))+E _(t)(r _(t) ,r _(t-1),

_(t))+E _(t)(l _(t) ,r _(t) ,l _(t-1) ,r _(t-1),

_(t))  Equation 16

The optimal sequence of labels and roles assignments may be determinedby mapping the energies to edge weights in a trellis graph, as furtherdescribed herein, and determine the shortest path through the graphusing dynamic programming.

Under some circumstances the assignments of labels and roles to playerdetections may be ambiguous. For example, when a player crosses the pathof another player exchanging roles, the appropriate label or rolecorresponding to one or more (x, y) detections may be difficult todetermine. In addition, both players may appear to be out of formationduring the role exchange. Because a set of detections D_(t) may havemultiple feasible assignments of labels and roles, a sequence of optimalper-frame solutions may result in an uninformative solution that differssignificantly from frame to frame, leading to assignments that rapidlyflicker among two or more possibilities between successive detectionframes. To counteract this effect, an approximately optimal sequence maybe determined by searching for a consistent temporal configuration oflabels and roles assignments among the top k independent solutionsproduced for each time instant.

The quantity N!^(2T) of possible sequences of labels and rolesassignments increases significantly as the number of players and timesteps increases. For example, for a ten-player team, a single timeinstant Y_(t) includes approximately 1.3×10¹³ possible assignmentsequences. In practice, a relatively small quantity of these possibleassignments are feasible. As such, the temporal search space of possibleassignments may be pruned by identifying the top k feasible assignmentsof labels and roles at each time instant.

In an alternative approach, the assignment analysis system may assignonly roles without assigning labels or identities. The probability of anassignment of roles for a single time instant is given by E_(t)(r_(t),

_(t)). Because players may swap roles at any time, there is no directrelationship between roles assignments at time t and at time T>t.Accordingly, E_(t)(r_(t), r_(t-1),

_(t))=0.

FIG. 4 illustrates a formation 400, a set of player detections 410, anda formation-detection mapping 420, according to one embodiment of thepresent invention.

The formation 400 illustrates a “W-M” formation, so named because thesubformation of the upper five players resembles the letter “W,” and thesubformation of the lower five players resembles the letter “M.” Thepositions of the players in the formation 400 are shown in an initial ordefault position, and may match the positions shown for this formation400 in a corresponding code-book.

The set of player detections 410 shows the positions of the ten playerson the field. The set of player detections 410 may come from a playerdetection system, as described herein. The set of player detections 410may include positional data only, without a mapping of the playerdetections to either the label/identity or the role of each player.

The formation-detection mapping 420 shows the mapping of the players inthe formation 400 to the set of player detections 410. As furtherdescribed herein, players are assigned to roles based on a probabilitymodel and the formation 400.

A formation 400 may be defined as a planar graph

=(

, E), where each vertex represents a particular role

, and where triangulated edges E between vertices explicitly modelspatial relationships.

For example, the left forward could be defined as the player to the leftof the center forward and in front of the left midfielder. Accordingly,if the above description is true, the triangle formed by [LF, CF, LM]would have an area of greater than zero. During play, the graph of theformation 400 deforms elastically as players move around the field, butwhere players maintain the formation 400.

If the shape of the formation is defined in terms of player labels oridentities, the graph of the formation 400 may exhibit internal twistsas players exchange roles during play. As such, the formation 400 wouldno longer be in canonical form. As a result, the analysis may be unableto track players based on prior data, other than local kinematicconstraints based on how much each player may move between successivedetection frames. On the other hand, if roles are identified, ratherthan labels, then the graph of the formation 400 may remain planar, inthat the formation 400 exhibits no intersecting edges.

Because roles are defined by spatial properties and relationships amongplayers, the assignment analysis system may infer player roles based ona set of player detections 410 that includes a set of (x, y) playerdetection locations. Three contextual cues may be used to determine anoptimal assignment of roles to detections, namely absolute (abs)context, relative (rel) context, and neighborhood (nbr) context, asshown in Equation 17 below:E(r _(t),

_(t))=E ^(abs)(r _(t),

_(t))+E ^(rel)(r _(t),

_(t))+E ^(nbr)(r _(t),

_(t))  Equation 17

Absolute context and relative context evaluate the likelihood of aspecific role being assigned to a particular detection, whileneighborhood context evaluates the feasibility of the combination of theindividual assignments. Each of these three contexts is described inturn.

With the absolute context, the assignment analysis system learns the 2Dprobability function P(

(i)|

_(t)(j)) of an (x, y) detection location having a particular role basedon manually-labeled exemplar data. The playing surface is divided into acollection of discrete cells, and frequency counts are generated thattabulate how often each role occupies each cell. Using these frequencycounts, the probability for each individual role assignment occupyingeach (x, y) location in

_(t) may be computed. The results for all roles and all field locationsmay be combined to generate an energy map that reflects the probabilityof the assignment r_(t) of roles. The energy function of the absolutecontext is given by Equation 18 below:

$\begin{matrix}{{E^{a\;{bs}}\left( {r_{t},{??}_{t}} \right)} = {{- \lambda^{{ab}\; s}}{\sum\limits_{i}^{\;}{\log\;{P\left( {\mathcal{R}(i)} \middle| {{??}_{t}\left( {{\hat{r}}_{t}(i)} \right)} \right)}}}}} & {{Equation}\mspace{14mu} 18}\end{matrix}$

With the relative context, all other locations {

_(t)\

_(t)(j)} are expressed in coordinates relative to

_(t)(j). As such, t player roles are defined by relative locations. Forexample, a left forward could be defined as the player in front of andto the left of all other players. Equivalently, all roles other than theleft forward would have a relative position that is behind and to theright of the left forward.

A descriptor G(

_(t)(j)) may be computed for each detection (x, y)_(j) in

_(t) by computing the locations of the remaining detections in

_(t) relative to (x, y)_(j). The relative displacements may then becoarsely quantized in terms of distance and angle. A threshold distancemay be selected, below which relative distance may be deemed to beinsignificant. For example, a threshold of three meters may be selected.As a result, if two players are detected within three meters of eachother, the relative distance of the two players would be deemed to bezero. Accordingly, given a reference player, other players would besorted into five bins relative to the reference player: within threemeters, in front, behind, left, and right. An exemplar descriptor G(

(i)) could be learned for each role using the manually-labeled trainingdata. The cost of assigning a particular role to a detection would bebased on the distance between the descriptor generated for the (x,y)_(j) the learned exemplar descriptor for the hypothesized role. Theenergy function of the relative context is given by Equation 19 below:

$\begin{matrix}{{E^{rel}\left( {r_{t},{??}_{t}} \right)} = {\lambda^{rel}{\sum\limits_{i}^{\;}{\mathbb{d}\left( {{G\left( {\mathcal{R}(i)} \right)},{G\left( {{??}_{t}\left( {{\hat{r}}_{t}(i)} \right)} \right)}} \right)}}}} & {{Equation}\mspace{14mu} 19}\end{matrix}$

With the neighborhood context, spatial relationships among the playersare encoded using a triangular mesh. The roles of the players aredefined through spatial relationships that exhibit particular patternsin the triangular mesh. An assignment of roles may maintain thesepatterns. Although the formation 400 may deform elastically as theplayers move on the field, an assignment of roles may maintain theplanarity of the triangular mesh such that no edges of the triangularmesh intersect.

Because a planar graph has no intersecting edges, a neighborhood contextenergy function may be defined that is proportional to the number ofedge intersections induced by a particular assignment of roles.

However, players may occasionally deviate from the formation 400, suchas when three players spread out to create a line. To achieve atemporally consistent solution over long time periods of play, theassignment analysis system may allow the assignment of roles to slightlyviolate the planar constraint. Accordingly, an alternative test ofplanarity may be used that measures triangle overlap area rather than acount of edge intersections.

FIG. 5 illustrates a set of formations with varying levels of areaoverlap, according to one embodiment of the present invention. As shown,the set of formations includes a formation with no overlap 500, aformation with small overlap 510, and a formation with large overlap520.

The formation with no overlap 500 illustrates an assignment of rolesthat results in a triangulation with no triangle overlap. If none of thetriangles overlap, then the graph of the formation is said to be planar.

However, if roles are incorrectly assigned, or if players get slightlyout of position, two or more triangles may overlap. While the quantityof edge intersections is an integer value, the area of overlap among alltriangles is a continuous measure. The formation with small overlap 510may result from cases where a player is slightly out of position. Theformation with large overlap 520, by contrast, may result from a highlyunlikely role assignment that does not preserve the expected spatialrelationship among roles. The energy function of the neighborhoodcontext is given by Equation 20 below:

$\begin{matrix}{{E^{nbr}\left( {r_{t},{??}_{t}} \right)} = {\lambda^{nbr}{\sum\limits_{{({a,b})} \in \Delta}^{\;}{{\Delta_{a}\bigcap\Delta_{b}}}}}} & {{Equation}\mspace{14mu} 20}\end{matrix}$where Δ_(a) and Δ_(b) represent a pair of triangles in the formation

.

The energy functions of absolute and relative context result in linearassignment problems, which may be efficiently solved using Hungarianalgorithm, as described herein. The neighborhood context evaluatessimultaneous role assignments for groups of six players, in that eachtriangle in the formation

is defined by three vertices, where each vertex is a role. As such,computing the overlap of two triangles involves six simultaneous roleassignments. The Hungarian algorithm is unable to such problems, and istherefore not suitable for computing neighborhood context costs.

Typically only a relatively small quantity of the N! possiblepermutations are feasible, where a feasible permutation represents anelastic deformation of

exhibiting little or no triangle overlap. As such, a “best assignments”approach may be employed to identify a likely subset of permutations byonly considering absolute and relative context, as shown in Equation 21below:

$\begin{matrix}\left\{ {{\hat{r}}_{t}^{1},{\hat{r}}_{t}^{2},{{\ldots\mspace{14mu}\left\{ {\hat{r}}_{t}^{k} \right\}} = {{\overset{k}{\underset{r_{t}}{\arg\;\min}}\;{E^{{ab}\; s}\left( {r_{t},{??}_{t}} \right)}} + {E^{rel}\left( {r_{t},{??}_{t}} \right)}}}} \right. & {{Equation}\mspace{14mu} 21}\end{matrix}$

Then, the subset of feasible role assignments {{circumflex over(r)}_(t)} is enumerated, and the neighborhood context is computed forthe feasible role assignments to determine a more optimal roleassignment.

In another alternative approach, the assignment analysis system mayassign only labels or identities without assigning roles. Theprobability of a particular assignment l_(t) of labels at time t may begiven by E_(t)(l_(t),

_(t)). If the observed data contains no appearance information, then theenergy function is a constant, and the assignment of labels is, ineffect, random. As such, the label ordering for the first frame may bedefined as the original ordering of the player detections L₀

l. However, because players have mass, the movements of human playersare governed by inertia, such that a label can only move so far from onedetection frame to the next. Accordingly, a permutation T_(t,T) may beinferred that preserves the arbitrary label ordering l_(t) establishedat time=t to a consistent label assignment l_(T) for a later time T>t,as given by equations 22 and 23 below:T _(t,T) Ĩ _(T) =Ĩ _(T)  Equation 22T _(t,T) =L _(t) L _(T) ⁻¹  Equation 23indicating that the labels, or identities, of players do not changeduring from time t to a later time T>t, such that a permutation matrixexists that restores the original identity ordering of the players.

Typically, T_(t,T) is estimated by determining how well observations attime T fit hypothesized motion models generated from previousdetections. The motions of players are assumed to be independent. Thatis, the trajectory of one player is not influenced by the trajectory ofanother player. At relatively high detection frame sampling rates, theplayer detections may be sufficiently fast to infer that thedisplacement of a player between two adjacent detection frames is nearzero, as expressed in Equation 24 below:

$\begin{matrix}{{E_{t}\left( {l_{t},l_{t - 1},{??}} \right)} = {\sum\limits_{i}^{\;}{{{{??}_{t}\left( {{\overset{\sim}{I}}_{t}(i)} \right)} - {{??}_{t - 1}\left( {{\overset{\sim}{I}}_{t - 1}(i)} \right)}}}^{2}}} & {{Equation}\mspace{14mu} 24}\end{matrix}$

As described above, the detections can be sorted by label/identity orrole. As such, there is a permutation Q_(t) may be defined thatrearranges the labels/identities into role order, as given by Equation25 below:Q _(t)

R _(t) L _(t) ⁻¹  Equation 25

During the course of the game, players may swap roles by crossing pathswith each other. As such, a matrix S_(t,T) may be defined that encodesthe role swaps that occur in the time period from time t to time T>t, asgiven by Equations 26 below:S _(t,T) Q _(T) =Q _(t) ∴S _(t,T) =Q _(t) Q _(t) ⁻¹  Equation 26

Role swapping between players is relatively infrequent, such thatplayers tend to keep the same role from one detection frame to the nextdetection frame. As a result, the matrix S_(t,T) may be substantiallythe same as the identity matrix I, such that S_(t,T)≈I. Accordingly,most of the off-diagonal elements of S_(t,T) are likely to be zero. Evenso, the off-diagonal elements of S_(t,T), may exhibit structure, in thatcertain groupings of roles are more likely to swap than others. Forexample, in field hockey, the left forward and right forward couldfrequently swap roles to create confusion for the defense. Similarly,the backs and mid-fielders could swap roles laterally as well, althoughlikely to a lesser degree. In addition, some players could also tend toswap in forwards-backwards directions, such as a left mid-fielderswapping roles with a left back. On the other hand, it would berelatively unlikely for a left forward to swap with a right defender.

As expressed in Equations 22 and 26, the assignments of both labels androles have temporal relations to the respective previous assignments oflabels and roles. Because role swapping is temporally sparse, S_(t,T)has an expected structure. By contrast, the tracking matrix T_(t,T) doesnot have an equivalent prior because the permutation matrices L_(t) andR_(t) implicitly incorporate the arbitrary detection order under whichthe (x, y) detections were observed at time t. Given the probability ofa particular S_(t,T) an equivalent expectations may be placed on Q_(T)that effectively correlates the assignments of labels l_(T) and rolesr_(T). Equation 22 and Equation 25 may be substituted into Equation 26to estimate S_(t,T) from role assignments and tracking, as shown inEquation 27 below:S _(t,T) R _(T) =R _(T) L _(t) ⁻¹ L _(T)S _(t,T) R _(T) =R _(t) L _(t) ⁻¹ L _(T)S _(t,T) R _(T) =R _(t) L _(t) ⁻¹(L _(t) T _(t,T) ⁻¹)S _(t,T) =R _(t) T _(t,T) ⁻¹ R _(T) ⁻¹  Equation 27

Equation 27 may be used to evaluate the difference between the currentrole assignments r_(T) and the role assignments achieved by propagatingthe previous role assignments r_(t) forward using the tracking results.Note that the final term in the energy function shown in Equation 16evaluates the same property, namely, the likelihood of transitioningfrom a previous simultaneous assignment (l_(t-1), r_(t-1)) of labels androles to a new simultaneous assignment (l_(t), r_(t)). As such, the costfor a particular role swapping {tilde over (s)}_(t-1,t)ε

_(n) determined by (l_(t-1), r_(t,1), l_(t), r_(t)) may be computed fromthe appropriate elements of the empirical model Ŝ_(t-1,t), as given byEquation 28 below:

$\begin{matrix}{{E_{t}\left( {l_{t},r_{t},l_{t - 1},r_{t - 1},{??}_{t}} \right)} = {- {\sum\limits_{i}^{\;}{\log\;{{\hat{S}}_{{t - 1},t}\left( {i,{{\overset{\sim}{s}}_{{t - 1},t}(i)}} \right)}}}}} & {{Equation}\mspace{14mu} 28}\end{matrix}$where {tilde over (s)}_(t-1,t) is the cost of swapping a role betweent−1 and t, and Ŝ_(t-1,t) is the model that defines the cost of swappinga role between t−1 and t.

In some cases, resolving the isolated labels and roles assignmentoptimizations may produce multiple ambiguous solutions. In addition,when the likelihood of role swapping is considered, the optimalsimultaneous assignments of labels and roles involving two non-optimalisolated assignment solutions. Because the potential solution space of Yis large, and the majority of simultaneous role and label/identitypermutations are unlikely, the top k permutations from the solutionspace for isolated tracking assignment T_(T)={T_(T) ¹, T_(T) ², . . . ,T_(T) ^(k)} and role assignment R_(T)={R_(T) ¹, R_(T) ², . . . , R_(T)^(k)}, where the top k permutations have a higher likelihood ofproviding the optimal assignment. A minimum energy configuration y_(t)*may then be determined for each time instant by evaluating Equation 28over all k×k proposed solutions. This approach may be extended tomultiple frames by mapping the problem onto a trellis graph 600, asfurther described herein.

FIG. 6 illustrates a trellis graph 600 that represents role optimizationfor multiple detection frames, according to one embodiment of thepresent invention. As shown, the trellis graph includes a source node610 and a sink node 620. As also shown, the trellis graph includesintermediate nodes arranged in a grid, where the vertical dimensionrepresents a potential assignment 650 and the horizontal positionrepresents a detection time 660. The nodes are connected via directededges, such as lowest cost path (LCP) edges 630.

For clarity, only a subset of potential assignments and detection timesare shown. The source 610 represents a beginning role and labelassignment for the team members. Each column represents a differentdetection time. Detection time 660(0) represents a detection time oft=1. Detection time 660(1) represents a detection time of t=2. Detectiontime 660(2) represents a detection time of t=3. The sink 620 representsan ending role and label assignment for the team members.

At each detection time 660, the trellis graph 600 includes a column ofnodes, where each node represents a vertex that is a potentialsimultaneous assignment of roles and labels. If the quantity ofpotential role assignments has been narrowed to k best permutations, andthe quantity of potential label assignments has also been narrowed to kbest permutations, then the trellis graph 600 includes up to k×kvertices, or potential role and label assignments, at each detectiontime 660. As shown, permutation 650(0) is the first potentialpermutation, permutation 650(1) is the first potential permutation, andpermutation 650(2) is the first potential permutation. Permutation650(3) is the ((k×k)−1)^(th) potential permutation, and permutation650(4) is the (k×k)^(th) potential permutation. Each directed edge, suchas lowest cost permutation (LCP) edges 630, represent a cost oftransition from a previous assignment at time=t−1, to a currentassignment at time=t. The least cost path from the source 610 to thesink 620, as represented by the LCP edges 630(0), 630(1), 630(2),630(3), indicate the optimal sequence of label and roll assignments asplay progresses.

A vertex at position v in a column of the trellis graph 600 at time trepresents a particular set of role and label assignments as defined byequation 29 below:V _(t) ^(v)≡(l _(t) ,r _(t))_(v)  Equation 28

A directed edge from a vertex in position u at time t−1 to a vertex inposition v at time t, has a weight as given by Equation 29 below:w ^(u→v) =E(r _(t),

_(t))+E(l _(t-1) ,l _(t),

_(t))+E(l _(t-1) ,r _(t-1) ,l _(t) ,r _(t),

_(t))  Equation 28

Accordingly, edges emanating from the source 610 have a weight of E(r₁),while edges leading directly to the sink 620 have a weight of zero. Theleast cost path (LCP) 630, as measured by the sum of edge weights fromthe source 610 to the sink 620, determines the optimal sequence y* ofrole and label assignments that minimizes the function as shown inEquation 15.

FIG. 7 sets forth a flow diagram of method steps for assigning playersto detection information, according to one embodiment of the presentinvention. Although the method steps are described in conjunction withthe systems of FIGS. 1-6, persons of ordinary skill in the art willunderstand that any system configured to perform the method steps, inany order, is within the scope of the invention.

As shown, a method 700 begins at step 702, where the assignment analysissystem learns absolute context descriptors. In one embodiment, theabsolute context descriptors may include manually-labeled position datafrom prior games. At step 704, the assignment analysis system learnsrelative context descriptors. In one embodiment, the relative contextdescriptors may also include manually-labeled position data from priorgames. At step 706, the assignment analysis system defines at least oneexemplar formation. At step 708, the assignment analysis system receivesa snapshot that includes (x, y) positions for a set of players. At step710, the assignment analysis system calculates absolute cost dataassociated with the player positions, based on the absolute contextdescriptors. At step 712, the assignment analysis system calculatesrelative cost data associated with the player positions, based on therelative context descriptors. At step 714, the assignment analysissystem estimates the top k probable role assignments based on theabsolute cost data and the relative cost data. At step 716, theassignment analysis system calculates a neighborhood cost for eachprobable solution. At step 718, the assignment analysis system selectsthe role assignment associated with the lowest cost. The method 700 thenterminates.

FIGS. 8A-8B set forth a flow diagram of method steps for assigningplayers to detection information, according to another embodiment of thepresent invention. Although the method steps are described inconjunction with the systems of FIGS. 1-6, persons of ordinary skill inthe art will understand that any system configured to perform the methodsteps, in any workable order, is within the scope of the invention.

As shown, a method 800 begins at step 802, where the assignment analysissystem learns absolute context descriptors. At step 804, the assignmentanalysis system learns relative context descriptors. In one embodiment,the absolute context descriptors, the relative context descriptors, orboth the absolute context descriptors and the relative contextdescriptors may include manually-labeled position data based on priorgames. The absolute context descriptors may be in the same data setrelative context descriptors. Alternatively, the absolute contextdescriptors may be in the same data set relative context descriptors. Atstep 806, the assignment analysis system defines at least one exemplarformation. At step 808, the assignment analysis system learns roleswapping probabilities. In one embodiment, the role swappingprobabilities may include manually entered data.

At step 810, the assignment analysis system receives a temporal sequenceof snapshots that includes (x, y) positions for a set of players at eachof a sequential set of time steps. At step 812, the assignment analysissystem selects a snapshot within the temporal sequence. At step 814, theassignment analysis system calculates absolute cost data associated withthe player positions, based on the absolute context descriptors. At step816, the assignment analysis system calculates relative cost dataassociated with the player positions, based on the relative contextdescriptors. At step 818, the assignment analysis system estimates thetop k probable role assignments based on the absolute cost data and therelative cost data. At step 820, the assignment analysis systemcalculates a neighborhood cost for each probable solution.

At step 822, the assignment analysis system calculates tracking costs.At step 824, the assignment analysis system estimates the top k probableidentity assignments based on the tracking cost data. At step 826, theassignment analysis system determines whether there are additionalsnapshots to process. If there are additional snapshots to process, thenthe method proceeds to step 812, described above.

If, however, at step 826, there are no additional snapshots to process,then the method proceeds to step 828, where the assignment analysissystem constructs a trellis graph that includes the top k probable roleassignments and the top k probable identity assignments. As step 830,the assignment analysis system computes the shortest path in the trellisgraph. The method 800 then terminates.

The techniques described herein may be used in various applications tosupport live analysis or post-analysis of sporting events or otherevents involving teams exhibiting role-based behavior. Such applicationsinclude, without limitation, measuring specific team behavior,retrieving plays similar to a reference play, or measuring the frequencyof a given play of interest.

In a first example, a pattern of role-swaps could be used to measurespecific or unique team behavior under a given set of circumstances,such as open versus pressured shots toward the goal. The quantity ofrole-swaps could be different during an open shot, such as when no orfew defenders are nearby, versus a pressured shot, where multipledefenders are nearby. Role-swap patterns could be analyzed to determineif a given play is indicative of an open shot or a pressured shot.Role-swap patterns could be compared between a current game versus oneor more previous games as a technique to identify a team based onrole-swap behavior or to determine whether the role-swap patterns duringa current game are consistent with or different from the team'srole-swap behavior during prior games.

In a second example, processed tracking data from a current play couldbe used to retrieve similar plays from one or more prior games. Theretrieval of prior plays could be based on an input query to a to adatabase management system, where the input query includes the processedtracking data of the current play of interest, and one or more inputfeatures of the current play, where an input feature represents acharacteristic of the play. The database management system wouldretrieve prior plays that are consistent with or similar to theprocessed tracking data of the current play and that exhibitcorrespondence to the input features. Such input features could include,without limitation, the role-swap behavior of the players, the travelspeed of the ball, or the distance between players. The distance measurecould be based on technically feasible distance metric, including,without limitation, Euclidean distance (L₂ norm) or maximum distance(L_(∞) norm). This approach could enable automated, online retrieval ofsimilar plays during a live sporting event, such as could be desired tosupport game analysis. Such an approach could yield more accurate andtimely retrieval results versus retrieval based on manually annotatedplays based on keywords, such as “three point shot” or “half courtshot.”

In a third example, multiple plays could be grouped into a cluster,where each play in the cluster exhibits similar role-swapping behavior.A particular play of interest could be specified, where the play ofinterest would be identified based on a threshold, such as a minimumdistance between a first player role and a second player role. Othersplays that are similar to the play of interest could be grouped into acluster, based on the threshold. The number of plays in the clusterwould indicate how often the play of interest occurs in a current gameor in one or more previous games. Dividing the number of plays in thecluster by the total number of plays in a game would yield thepercentage occurrence of the play of interest.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder or out of order, depending upon the functionality involved. Itwill also be noted that each block of the block diagrams and/orflowchart illustration, and combinations of blocks in the block diagramsand/or flowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method for assigning roles to agents in a firstgroup of agents engaging in an activity, comprising: receiving a firstset of detections, wherein each detection in the first set of detectionscomprises a physical location; defining an exemplar formation comprisingan arrangement of each role in a set of roles; calculating a first costfunction between at least one detection in the first set of detectionsand at least one role in the set of roles; generating a first set ofpermutations based on the first cost function; and assigning a firstrole in the set of roles to a first detection in the first set ofdetections based on the first set of permutations.
 2. The method ofclaim 1, further comprising separating the first set of detections intoa first portion comprising physical locations associated with the firstgroup of agents and a second portion comprising physical locationsassociated with a second group of agents.
 3. The method of claim 1,wherein each physical location in the first set of physical locationscomprises an absolute location in a playing field configured for a teamsport.
 4. The method of claim 1, wherein each physical location in thefirst set of physical locations comprises a position of a first agent inthe first group of agents relative to a second agent in the first groupof agents.
 5. The method of claim 1, wherein assigning a first role inthe set of roles to a first detection in the first set of detectionscomprises: selecting a set of potential permutations for evaluation,wherein each permutation defines a relationship between each detectionin the set of detections and each role on the set of roles; for eachtime step in a set of time steps, calculating a transition cost totransition from a first permutation at a first time step to a secondpermutation at a second time step; selecting a lowest cost path throughthe set of time steps based on the transition costs; and assigning thefirst role in the set of roles to the first detection in the first setof detections based on the lowest cost path.
 6. The method of claim 1,wherein assigning a first role in the set of roles to a first detectionin the first set of detections comprises: selecting a set of potentialpermutations for evaluation, wherein each permutation defines arelationship between each detection in the set of detections and eachrole on the set of roles; for each time step in a set of time steps,calculating a transition cost to transition from a first permutation ata first time step to a second permutation at a second time step;selecting a lowest cost path through the set of time steps based on thetransition costs; determining that the lowest cost path is temporallyinconsistent with a current assignment of roles in the set of roles todetections in a second set of detections; selecting a next lowest costpath through the set of time steps based on the transition costs; andassigning the first role in the set of roles to the first detection inthe first set of detections based on the next lowest cost path.
 7. Themethod of claim 1, further comprising: calculating a second costfunction between at least one detection in the first set of detectionsand at least one agent in the first group of agents; generating a secondset of permutations based on the second cost function; and assigning afirst identity associated with a first agent in the first group ofagents to a first detection in the first set of detections based on thesecond set of permutations.
 8. The method of claim 7, furthercomprising: receiving a second set of detections, wherein each detectionin the second set of detections comprises a physical location;calculating a third cost function between at least one detection in thesecond set of detections and at least one role in the set of roles;generating a third set of permutations based on the third cost function;calculating a fourth cost function between at least one detection in thesecond set of detections and at least one agent in the first group ofagents; and generating a fourth set of permutations based on the fourthcost function; wherein assigning the first role to the first detectionis further based on at least one of the second set of permutations, thethird set of permutations, and the fourth set of permutations, andwherein assigning the first identity to the first detection is furtherbased on at least one of the first set of permutations, the third set ofpermutations, and the fourth set of permutations.
 9. The method of claim1, further comprising: determining that a first quantity of detectionsin the first set of detections is less than a second quantity of rolesin the set of roles; and selecting a second role in the set of rolesthat lacks a corresponding detection in the first set of detectionsbased on the first cost function.
 10. The method of claim 1, furthercomprising: determining that a first quantity of detections in the firstset of detections is greater than a second quantity of roles in the setof roles; and selecting a second detection in the first set ofdetections that lacks a corresponding role in the set of roles based onthe first cost function.
 11. The method of claim 1, wherein the exemplarformation is selected by a user from a predefined set of exemplarformations.
 12. The method of claim 1, further comprising characterizinga behavior of the first group of agents based at least in part on theassignment of the first role to the first detection, wherein thebehavior is defined by a plurality of sets of detections including thefirst set of detections.
 13. The method of claim 1, further comprising:retrieving a first sequence of detections based at least in part on theassignment of the first role to the first detection, wherein the firstsequence of detections comprises a plurality of sets of detectionsincluding the first set of detections, each set of detectionscorresponding to a different time; retrieving an input featurerepresenting a characteristic of the first sequence of detections;identifying a second sequence of detections included in a database ofdetection data based on the input feature; and retrieving the secondsequence of detections from the database.
 14. A non-transitorycomputer-readable storage medium including instructions that, whenexecuted by a processor, cause the processor to assign roles to agentsin a first group of one or more agents engaging in an activity, byperforming an operation comprising: receiving a first set of detections,wherein each detection in the first set of detections comprises aphysical location; defining an exemplar formation comprising anarrangement of each role in a set of roles; calculating a first costfunction between at least one detection in the first set of detectionsand at least one role in the set of roles; generating a first set ofpermutations based on the first cost function; and assigning a firstrole in the set of roles to a first detection in the first set ofdetections based on the first set of permutations.
 15. Thenon-transitory computer-readable storage medium of claim 14, furthercomprising separating the first set of detections into a first portioncomprising physical locations associated with the first group of agentsand a second portion comprising physical locations associated with asecond group of agents.
 16. The non-transitory computer-readable storagemedium of claim 14, wherein each physical location in the first set ofphysical locations comprises an absolute location in a playing fieldconfigured for a team sport.
 17. The non-transitory computer-readablestorage medium of claim 14, wherein each physical location in the firstset of physical locations comprises a position of a first agent in thefirst group of agents relative to a second agent in the first group ofagents.
 18. The non-transitory computer-readable storage medium of claim14, wherein assigning a first role in the set of roles to a firstdetection in the first set of detections comprises: selecting a set ofpotential permutations for evaluation, wherein each permutation definesa relationship between each detection in the set of detections and eachrole on the set of roles; for each time step in a set of time steps,calculating a transition cost to transition from a first permutation ata first time step to a second permutation at a second time step;selecting a lowest cost path through the set of time steps based on thetransition costs; and assigning the first role in the set of roles tothe first detection in the first set of detections based on the lowestcost path.
 19. The non-transitory computer-readable storage medium ofclaim 14, wherein assigning a first role in the set of roles to a firstdetection in the first set of detections comprises: selecting a set ofpotential permutations for evaluation, wherein each permutation definesa relationship between each detection in the set of detections and eachrole on the set of roles; for each time step in a set of time steps,calculating a transition cost to transition from a first permutation ata first time step to a second permutation at a second time step;selecting a lowest cost path through the set of time steps based on thetransition costs; determining that the lowest cost path is temporallyinconsistent with a current assignment of roles in the set of roles todetections in a second set of detections; selecting a next lowest costpath through the set of time steps based on the transition costs; andassigning the first role in the set of roles to the first detection inthe first set of detections based on the next lowest cost path.
 20. Acomputing system, comprising: a memory that is configured to storeinstructions for a program; and a processor that is configured toexecute the instructions for the program to assign roles to agents in afirst group of one or more agents engaging in an activity, by performingan operation comprising: receiving a first set of detections, whereineach detection in the first set of detections comprises a physicallocation; defining an exemplar formation comprising an arrangement ofeach role in a set of roles; calculating a first cost function betweenat least one detection in the first set of detections and at least onerole in the set of roles; generating a first set of permutations basedon the first cost function; and assigning a first role in the set ofroles to a first detection in the first set of detections based on thefirst set of permutations.